Skip to content

[ROCm] Fix intranode queue ordering for shared buffers#18

Open
AndreasKaratzas wants to merge 1 commit into
ROCm:mainfrom
AndreasKaratzas:akaratza_fix_rocm_intranode_ordering
Open

[ROCm] Fix intranode queue ordering for shared buffers#18
AndreasKaratzas wants to merge 1 commit into
ROCm:mainfrom
AndreasKaratzas:akaratza_fix_rocm_intranode_ordering

Conversation

@AndreasKaratzas

Copy link
Copy Markdown
Contributor

The Distributed Tests / DBO test group on ROCm vLLM CI fails in build 11147 when successive DBO microbatches share one ROCm DeepEP buffer. The intranode path can publish queue progress without a complete producer/consumer ownership chain, while whole-workgroup barriers can pair unrelated per-rank rounds. This change uses per-rank wave barriers and release/acquire ordering for dispatch and combine tail publication, queue-slot reuse, receiver progress, retirement, and shared tail fan-out. It also adds a two-rank regression test that drives asymmetric and randomized microbatches through repeated queue wraparound on the same buffer.

@AndreasKaratzas
AndreasKaratzas marked this pull request as ready for review July 24, 2026 00:13
@kudomcho

Copy link
Copy Markdown
Collaborator

Do you have test suite results after this change?

@AndreasKaratzas

Copy link
Copy Markdown
Contributor Author

@kudomcho For vLLM you mean?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants